Peta-Scale Embedded Photonics Architecture for Distributed Deep Learning Applications

As Deep Learning (DL) models grow larger and more complex, training jobs are increasingly distributed across multiple Computing Units (CU) such as GPUs TPUs. Each CU processes a sub-part of the model synchronizes results with others. Communication among these CUs has emerged key bottleneck in process. In this work, we present SiPAC, Silicon Photonic Accelerated Compute cluster. SiPAC accelerates DL by means two co-designed components: photonic physical layer novel collective algorithm. The exploits embedded photonics to bring peta-scale I/O directly optimized cluster uses resonator-based optical wavelength selectivity realize hardware multi-casting. algorithm builds on multi-casting primitive. This combination expedites variety communications commonly employed potential drastically ease communication bottlenecks. We demonstrate feasibility realizing architecture through 1) an testbed experiment where array comb laser wavelengths shuffled cascaded ring switch, each selecting forwarding increase effective bandwidth hence demonstrating multicasting primitive, 2) four-GPU running realistic workload that achieves 22% system-level performance improvement relative similarly-sized leaf-spine topology. Large scale simulations show 1.4× 5.9× time reduction compared state-of-the-art compute clusters for representative communications.